Repository navigation
FEAT: Save targets created through the API (part 8) - #4026
Conversation
The target catalog now applies the path rule to required parameters too, so a target that needs a disallowed path is left out instead of listed without the field, matching the scorer catalog. allowed_path_params is typed frozenset[str] so a bare string cannot turn the allow-list into a substring match.
The target catalog still lists a target whose required path parameter is its server-set upload folder when the server has one, and leaves it out otherwise.
# Conflicts: # tests/unit/backend/test_converter_service.py
…benchmark's recipes folder
A suite launch runs its launch scripts during preflight and sends its scans without them, so the clash is raised there. Map it to 409 with its message, as a single launch does, and drop the per-entry message that a clash can no longer reach.
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
Please keep the useful persist/origin feature, but move its reusable persistence operation into the registry library before inclusion. The new _prepare_saved_params_async, _refuse_temporary_references, and _save_and_register_async helpers in pyrit/backend/services/target_service.py:738-876 own durable-parameter preparation, which registered dependencies may survive a restart, and save/register/rollback. The registry currently exposes the storage primitives, so a notebook caller wanting to save the same target must depend on backend service/request types or reconstruct those rules and compensation steps. The library-first framework guidance in #4175 assigns this behavior to the library and keeps backend services as adapters. Extract this operation using the existing registry, recipe models, rules and storage; leave authenticated-owner/configured-source extraction, HTTP error mapping, and runtime/process coordination in the backend. No generic persistence framework or redesign of the already-reviewed predecessor is needed.
This inclusion recommendation was independently verified at 48bc6d14c756d00990dc8ff4b6ab09bcaa0d1828. Implementation review is paused at that gate; this is not a full source review or approval.
# Conflicts: # pyrit/prompt_target/common/prompt_target.py # pyrit/prompt_target/hugging_face/hugging_face_chat_target.py
# Conflicts: # tests/unit/backend/test_runtime_lifecycle.py
|
Thanks, that makes sense. Saving now lives in the registry library ( I made the same move up the stack, so converters, scorers, techniques and copies save through the library too, and the backend only keeps the lock that orders saves and deletes. |
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
Reviewed this head with complete source coverage and independent finding verification. Five new issues remain below; the existing rollback concern also remains open. The main blocker is that the public Python save workflow accepts recipes that the same-head restore path cannot rebuild. No approval or source changes.
Saving now refuses what restore can't rebuild: a setting left to the server's environment (each target type declares these, replacing the backend's class-name table) and parameters the create API can't set. A cancelled save finishes or is undone before the cancellation goes on, rollback deletes only the version this save wrote, storage failures log a redacted cause chain, and capability detection keeps the target's origin.
Saved-entry storage failures now pass their traceback through redact_urls, and code scanning flags the scheme pattern as slow on a long run of letters. A scheme is short, so bounding it keeps the match linear.
Richard Lundeen (richlundeen)
left a comment
There was a problem hiding this comment.
Automated review completed. No blocking concerns remain. This review records source coverage; merging is handled separately.
Description
This is one of the changes for saving Registry entries across restarts. It builds on #4025 (merged), which refuses saved targets that would authenticate with a key.
Problem. A target created through
POST /api/targetsis gone when the backend restarts. Restore (#3909) brings saved entries back at every start, but nothing saves one yet, and the target list doesn't say which targets will come back.Change.
Saving is a registry library operation, so Python code saves a target the same way the API does. In
pyrit/registry/registry_recipe_save.py(exported frompyrit.registry),prepare_saved_params_asyncturns off the values a target would take from this server and refuses what a saved entry can't hold, using the same checks restore does, including which targets it uses come back after a restart. It also accepts only what restore's build accepts: a setting the target would read from this server's environment, such as an OpenAI model name, has to be set, and a parameter the create API can't take, such ashttpx_client_kwargs, is refused (400) before anything is written.save_then_register_asyncwrites the recipe only if the name is free, registers the entry as saved, and deletes the recipe again if registering fails, but only the version it wrote, so a file another writer saved since is left alone. A save that's cancelled finishes, or is rolled back, before the cancellation goes on. The backend only passes the signed-in owner and the configured source, builds the response, and maps the library's errors to 400, 409 and 503.POST /api/targetstakes"persist": true, which needsregisterleft on and a name that starts with a lowercase letter and has at most 64 lowercase letters, digits and underscores (otherwise 422). The target is built as before, saved as a recipe underregistry_recipes_source(by default~/.pyrit/registry_recipes, so no Azure is needed) with the signed-in user recorded as the one who saved it, and then registered. If registering fails, the file is deleted again; if that delete fails too, the request gets 503 and the server log names the target and its folder, so the file can be removed.Before saving, an OpenAI target's empty
headersis set to"{}", so a saved one doesn't send this server'sOPENAI_ADDITIONAL_REQUEST_HEADERS. Then the request goes through the same check restore uses, so a target that would hold or use a key, or leave an address to this server's settings, gets 400, and the message says it can still be created without saving it. A saved target can use saved targets and targets from configuration, which keep their own sign-in, but not a Temporary target, such as a round-robin member that won't come back after a restart.Each target type declares the settings it reads from this server's environment when they're empty (
environment_backed_parametersonPromptTarget). That replaces the backend's table keyed by class name, soGET /api/targets/typesmarks the same fields required that a save checks.A name that a registered target or a saved file already has gets 409. The file is written create-only, so two saves of one name save one target. Storage that can't be written gets 503 and the target isn't registered. The server log names the target and its folder, with the error behind it and its chain (URL credentials removed, as restore logs), in case the write reached storage anyway. A failed rollback is logged the same way.
A target type that a scan's launch scripts added, or put in place of another class under an existing name, can't be saved (400): restore couldn't rebuild it after a restart, and with the backend staying not ready until every saved entry restores, that file would keep it down.
Each listed target has an
origin:saved,configuration(registered by the server's startup scripts), ortemporary(gone after a restart or reinitialization, such as a target created withoutpersistor registered by a scan's launch scripts). Capability detection returns it too.The seeded browser end-to-end runs, and the backend end-to-end runs when they start their own backend, point
registry_recipes_sourceat a new temp folder, so they don't read or write the targets saved on the machine running them.What stays the same. Creating a target without
persistworks as before, and permission checks are unchanged and come first, so a non-admin whose target would use a server credential still gets 403. A save runs inside the create request, and apply already reports busy while a request is in flight, so no new lock is needed. Restore is unchanged, except that it first marks the targets the startup scripts registered asconfiguration.Not covered.
GET /api/config/runtime. That can still happen after an upgrade or a change to the environment.Tests and Documentation
tests/unit/registry/test_registry_recipe_save.py(no backend imports): a target built from prepared parameters is saved, with"headers": "{}", and registered as saved; a typed key, an empty key, a header or URL that can carry a credential, a class a launch script added, and a Temporary target it uses are refused, also when the class's metadata isn't cached; the recipe is deleted when registering fails, and only a taken name is a conflict; a setting left to the server's environment and a parameter the API can't take are refused; storage failures are reported as unavailable, and the log names the target and its folder with the cause, URL credentials removed; a failed undo is reported as storage trouble and logged with its cause; rollback leaves a file another writer replaced; a cancelled save finishes registering, or rolls back, before the cancellation goes on.tests/unit/backend/test_target_service.py: an Entra ID target is saved and restore rebuilds it; a no-key target is saved without an owner when sign-in is off; a target that can't be saved is neither created nor saved; a taken name is a conflict only when saving; a saved file with the name is a conflict, stays as it was, and the built target is released; two saves of one name save one target; a round-robin is saved only over targets that come back after a restart; a class registered under another name is saved so restore finds it; a reserved name is refused as invalid, not as taken; the list shows where each target comes from; the administrator gate refuses an Entra ID Blob target before construction, with and withoutpersist, and nothing is registered or written.tests/unit/backend/test_api_routes.py: a save conflict maps to 409 and a storage failure to 503; the saved-entries source and the signed-in owner (or none when sign-in is off) reach the service;persistneedsregisterand a valid name.tests/unit/backend/test_mappers.py: the origin of each registered target, and on the instance.tests/unit/backend/test_target_detection.py: detection returns a saved target's origin, and none for its inner targets.tests/unit/backend/test_runtime_lifecycle.py: startup marks the targets its scripts register but not saved ones; a save in flight makes apply report busy.tests/unit/registry/test_target_registry.py: a class a launch script adds, or puts in place of another under an existing name, is temporary.tests/unit/registry/test_instance_registry.pyandtest_registry_recipe_storage.py: startup entries are marked and keep the mark when replaced; a recipe is created once, lists back with its owner and version, and is deleted once.python -m pytest -n 8 tests/unit/converter tests/unit/prompt_target tests/unit/registry tests/unit/backend: 11,883 passed, 9 skipped, 4 xfailed. The cancellation behavior was also checked on Python 3.11 and 3.14, includingasyncio.timeoutturning the passed-on cancellation intoTimeoutError.npm run lintandnpm run type-check(forplaywright.config.ts): passed.E2E_SEEDED_MODE=true npx playwright test --project=seeded --workers=1: 108 passed, with the backend started from the generated temp config.main's new RUF and F821 rules), ty, and the repo's keyword-only, async-suffix, public-import and docs checks: passed.httpx_client_kwargsgets a plain 400 rather than a can't-be-saved one, and one recipe file is written.saved); the same name gets 409; a typed key, an empty key and an endpoint with?sig=get 400 without echoing the key; exactly one recipe file is written, with no key. After a restart the saved target is back and answers a chat message, and the Temporary one is gone.pyrit/backend/README.mdanddoc/getting_started/pyrit_conf.md: how to save a target, the name rule, what it can hold, theheadersdefault, the OpenAI SDK settings that still apply, and the status codes.No JupyText changes.